Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of representations while searching them efficiently. We conduct extensive experiments showing that our approach significantly improves the perplexity of pre-trained language models on new distributions, and can scale efficiently to much larger contexts than previously proposed local cache models.

Finally, we integrate the results from both branches to achieve WSI classification.

Neural Information Processing Systems http://nips.cc/

Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of representations while searching them efficiently. We conduct extensive experiments showing that our approach significantly improves the perplexity of pre-trained language models on new distributions, and can scale efficiently to much larger contexts than previously proposed local cache models.

Neural Information Processing Systems http://nips.cc/

Training large-scale image recognition models is computationally expensive.

Finally, we integrate the results from both branches to achieve WSI classification.

This paper discusses an extensions to the recently proposed continuous cache models by Grave et al. The authors propose a continuous cache model that is unbounded, hence can take into account events that happened an indefinitely long time ago. While interesting, the paper fails to provide good experimental evidence of its merits. Its main statement is that this model is better than Grave et al., but then does not compare with it. It only seems to compare with cache models from the nineties (Kuhn et al.), although that is not even clear as they spend only one line (line 206) discussing the models they compare with.

This paper presents a cache model to be used in image recognition tasks. The authors argue that class specific information can be retrieved from earlier layers of the network to improve the accuracy of an already trained model, without having to re-train of finetune. This is achieved by extracting and caching the activations of some layers along with the class at training time. At test time a similarity measure is used to calculate how far/close the input is compared to information stored in memory. Experiments show that performance is improved in CIFAR 10/100 and ImageNet.

Neural Information Processing Systems http://nips.cc/